Running head: A COMPARISON OF THE EXACT KRUSKAL-WALLIS A comparison of the Exact Kruskal-Wallis Distribution to Asymptotic Approximations for All Sample Sizes

We generated exact probability distributions for sample sizes up to 35 in each of three groups ( 105 N  ) and up to 10 in each of four groups ( 40 N  ). We provided a portion of these exact probability tables and compared the exact distributions to the chi-square, gamma, and beta approximations. The beta approximation was best in terms of the root mean squared error. At specific significance levels either the gamma or beta approximation was best. These results suggest that the most common approximation, the chi-square approximation, is not a good choice, though for larger total sample sizes and equal numbers in each group, any of these three approximations are reasonable. For sample sizes up to 105, we can now provide exact tables that negate the use of an approximation. A comparison of the Exact Kruskal-Wallis 3 A comparison of the Exact Kruskal-Wallis Distribution to Asymptotic Approximations for All Sample Sizes Up to 105 Kruskal and Wallis's (1952) rank-based test of location equality for three or more groups may be among the most useful of available hypothesis testing procedures for behavioral and social science research. It is also a relatively popular method. A recent search on APA PsycNet for “Kruskal-Wallis” returned 268 results whereas the search term “hierarchical linear models” returned only 233 results, after limiting the search to peer-reviewed journal articles published between 2000 and 2011. The popularity of the Kruskal-Wallis test may be attributed to its usefulness in a variety of disciplines such as education, psychology, and medicine. It is also suitable for, and has been applied to, a wide array of topics such as the validity of educational or psychological measures (Armstrong, MacDonald, & Stillo, 2010; Jang, Chern, & Lin, 2009; Rajasagaram, Taylor, Braitberg, Pearsell, & Capp, 2009; Tarshis & Huffman, 2007; Yin & Shavelson, 2008), teacher characteristics (Finson, Pedersen, & Thoms, 2006; Gömleksiz & Bulut, 2007), child development (Belanger & Desrochers, 2001), and adolescent behavior and learning disabilities (Plata & Trusty, 2005; Plata, Trusty, & Glasgow, 2005). Most applications of the Kruskal-Wallis test use a large-sample approximation instead of the exact distribution. Indeed, none of the articles previously cited report use of an exact test and almost all of them report use of the chi-square approximation even though other approximations exist. Statistical packages such as SPSS and R default to the chi-square approximation and only offer the exact distribution for very small sample sizes. Unfortunately, very little is known about the veracity of the chi-square and other approximations for total sample sizes beyond about fifteen participants. This limitation is largely due to the lack of exact probability tables for A comparison of the Exact Kruskal-Wallis 4 moderate to large sample sizes. To overcome this limitation, we recently extended the exact probability tables to a total sample size of 105 participants as explained below. The purposes of this paper are to (a) share a portion of our exact probability tables, and (b) examine three large sample approximations with respect to the exact distribution. The Kruskal-Wallis Test Parametric methods, along with the requirement for a stronger set of assumptions, continue to dominate the research landscape despite convincing studies that call into question the wisdom of making such assumptions (Micceri, 1989). Replacing original scores with ranks does not inherently lead to lower power, as one might suppose, but rather can result in a power increase at best and a slight power loss, at worst. This has been verified using both Pitman and finite efficiency indices (Hettmansperger, 1984). A second criticism of nonparametric procedures, in general, and rank-based procedures, in particular, is that critical values or p-values are either difficult to compute or that tables of critical values are limited. Unlike the power myth, this criticism has some basis in reality. For example, when Kruskal and Wallis (1952) introduced their test, they provided exact probability tables for samples with five or less in each of three groups. Obviously such tables have limited value, though the way out of this predicament is to derive approximations that can be used when the table size is exceeded. Kruskal and Wallis derived three such approximations based on the chi-square, incomplete-gamma, and incomplete-beta distributions. Most research on the Kruskal-Wallis statistic (H) has focused on comparing its performance to one-way analysis of variance (e.g. Boehnke, 1984; Harwell, Rubinstein, Hayes, & Olds, 1992) or evaluating its assumptions (e.g.Vargha & Delaney, 1998). Little has been done The complete tables for significance levels of .1, .05, and .01 span over 200 pages. They are available upon request from the authors. Complete exact distributions are also available but they require over 1 terabyte of disk storage. A comparison of the Exact Kruskal-Wallis 5 to study the efficacy of these approximations yet such study should be paramount because other types of studies, including those referenced above, rely on these approximations. In short, if the approximations for the percentiles of H are problematic, studies that use approximate instead of exact cumulative probabilities for H are suspect. It is really no surprise that there is a paucity of research on the large-sample approximations. Exact probability tables of the Kruskal-Wallis statistic have slowly progressed over the years. Kruskal and Wallis (1952) provided exact probability tables when they proposed their test, but their tables were limited to 15 N  where N is the total sample size. Iman, Quade, and Alexander (1975) published more extensive exact probability tables for H, but even for their tables, none of the sample sizes exceeded eight for any of the groups. More recently, exact probabilities have been computed for samples as large as 45 N  (Spurrier, 2003) and 60 N  (Meyer & Seaman, 2006). Commercial software for computing exact probabilities for nonparametric statistics is more limited than the existing probability tables. SPSS Exact Tests does not provide exact probabilities for H for sample sizes larger than 15 (Mehta & Patel, 2010) and we were unable to obtain exact cumulative probabilities for H from StatXact 4.0 (Cytel, 2000) with samples as small as 30 participants. The statistical package R (R Development Core Team, 2011) includes a Kruskal-Wallis test in its base package but this test uses the chi-square approximation. Documentation for the R add-on package muStat (Wittkowski & Song, 2010) suggests it can provide exact probabilities for the Kruskal-Wallis test but this assertion is only true when there are no more than two groups. As such, muStat does not compute exact probabilities in cases where the Kruskal-Wallis test is most useful; it does not compute exact probabilities when there are more than two groups. Our comprehensive review of commercially available software did not A comparison of the Exact Kruskal-Wallis 6 find any that provided exact critical or p values for even moderately sized samples. Indeed, even when software documentation claims to provide such values, the software resorts to Monte Carlo or theoretical distribution approximations for all but the smallest sample size arrangements. This is understandable given the resource-intensive nature of computing exact values, as we describe below. Exact probability values are necessary to study the veracity of approximations, yet approximations are only necessary when exact probability values do not exist. As the exact probability tables cover an increasing larger sample size, the chi-square, gamma, and beta approximations can be studied more rigorously to support inferences about their usefulness in settings that involve even larger sample sizes. Our purpose in this paper is to provide critical values for H with even larger samples and then to compare these with approximate values. At the time of this writing, we have created the exact probability distributions for sample sizes up to 35 participants in each of three groups ( 105 N  ) and 10 participants in each of four groups ( 40 N  ). We created the tables for all possible configurations of unequal and equal sample sizes. The H Statistic and Related Quantities Consider k independent samples from distributions with CDFs of 1 2 ( ), ( ), , ( ) k F x F x F x        , where i  is a location parameter for population i. We wish to know if there are differences in location among the k populations, so we can test the null hypothesis 0 1 2 : k H      (1) against the alternative 1 : for at least one . i j H i j     (2) A comparison of the Exact Kruskal-Wallis 7 This location parameter is general and merely denotes a shift of the otherwise common distribution functions. In practice, a test of the above null hypothesis is usually considered a test of median (or mean) equality and is therefore similar to the one-way analysis of variance (ANOVA) as a test of means. Unlike the ANOVA test, which requires a specific form of distribution identity, namely normal distributions, the Kruskal-Wallis test merely assumes continuous populations that might differ in location, rather than shape. Kruskal and Wallis (1952) derived a test of the above hypothesis using the H statistic which can be viewed as the nonparametric analog of the F statistic in this one-way design. Given 1, , i k   independent random samples, each with i n observations, all 1 k i i n N    observations are ranked together from lowest to highest. The Kruskal-Wallis H statistic is based on the sum of ranks for each sample, i R , and is given by,